GRADIENT DESCENT

Beyond Reptile: Meta-Learned Dot-Product Maximization between Gradients for Improved Single-Task Regularization (2021.findings-emnlp)

Challenge:	Existing pipelines for generative tasks require extensive manual effort and domain expertise to achieve task-optimal performance.
Approach:	They propose a framework bridging discrete and continuous prompt optimization through feedback-guided gradient descent in embedding space.
Outcome:	The proposed framework bridges discrete and continuous prompt optimization through feedback-guided gradient descent in embedding space.

Copied to clipboard

Akhil Kedia, Sai Chetan Chinthakindi, Wonho Ryu

Neural Event Semantics for Grounded Language Understanding (2021.tacl-1)

Challenge:	Existing approaches to improve generalization of neural models use a small component of the gradient for maximizing dot-product between batches.
Approach:	They propose to use a finite differences first-order algorithm to calculate a gradient from dot-product of gradients and regularize it.
Outcome:	The proposed method outperforms previous approaches of Reptile and MAML when used as a regularization technique.

Copied to clipboard

Shyamal Buch, Li Fei-Fei, Noah D. Goodman

Information Extraction & Text Mining

Challenge:	a new framework for compositional grounded language understanding is proposed . NES is trainable end-to-end by gradient descent with minimal supervision.
Approach:	They propose a conjunctivist framework for compositional grounded language understanding . they use words as classifiers that compose to form a sentence meaning by multiplying output scores .
Outcome:	The proposed framework improves on compositional grounded language tasks.

Meta-learning via Language Model In-context Tuning (2022.acl-long)

Copied to clipboard

Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, He He

When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario (2023.findings-acl)

Challenge:	Recent advances in large language models have reduced "task learning and prediction" to a simple sequence prediction problem.
Approach:	They propose a meta-learning method that recasts task adaptation and prediction as a sequence prediction problem.
Outcome:	The proposed method outperforms MAML on two classification tasks and improves on binaryClfs.

Copied to clipboard

Chengcheng Han, Liqing Cui, Renyu Zhu, Jianing Wang, Nuo Chen, Qiushi Sun, Xiang Li, Ming Gao

In-context Learning and Gradient Descent Revisited (2024.naacl-long)

Challenge:	Large pre-trained language models (PLMs) are expensive and may not be open-sourced due to commercial considerations and potential risks of misuse.
Approach:	They propose to introduce gradient descent into black-box tuning scenario . they propose a method which integrates gradient descent and derivative-free optimization .
Outcome:	The proposed method achieves significant performance gains over previous state-of-the-art methods.

Copied to clipboard

Gilad Deutch, Nadav Magar, Tomer Natan, Guy Dar

Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent (2021.emnlp-main)

Challenge:	In-context learning (ICL) has shown impressive results in few-shot learning tasks, yet its underlying mechanism remains elusive.
Approach:	They propose a simple gradient descent-based optimization procedure that respects layer causality and improves similarity scores significantly.
Outcome:	The proposed procedure improves similarity scores on untrained models despite not showing ICL.

Copied to clipboard

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah A. Smith

Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2022.acl-long)

Challenge:	Evidence is emerging that neural networks learn due to inductive bias in the training routine, typically a variant of gradient descent (GD).
Approach:	They propose to characterize GD as an inductive bias in transformer training . they document norm growth in transformer language models and show they are saturated .
Outcome:	Empirically, we document norm growth in the training of transformer language models . the results suggest saturation is a new characterization of an inductive bias implicit in GD .

Copied to clipboard

Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, Maosong Sun

Synthetic Data Made to Order: The Case of Parsing (D18-1)

Challenge:	Recent studies suggest that pre-trained language models have gained rich knowledge during pre-training.
Approach:	They propose to tune pre-trained language models with task-specific prompts to improve and stabilize prompttuning.
Outcome:	Extensive experiments on zero and few-shot text classification tasks show that prompt-tuning improves and stabilizes prompttun-ing.

Copied to clipboard

Dingquan Wang, Jason Eisner

Syntactic Text Processing

Challenge:	supervised dependency parsing is a core task in natural language processing, but unsupervised parsers can hardly produce useful parses.
Approach:	They propose to permute the constituents of an existing dependency treebank so that its surface part-of-speech statistics approximately match those of the target language.
Outcome:	The proposed method improves the parsing accuracy of a target language . the proposed method is based on a distribution of gold POS bigrams .

Non-Parametric Adaptation for Neural Machine Translation (N19-1)

Copied to clipboard

Ankur Bapna, Orhan Firat

Machine Translation

Challenge:	Neural Networks trained with gradient descent are susceptible to catastrophic forgetting due to parameter shift during the training process.
Approach:	They propose a semi-parametric approach that relies on local phrase level similarities to retrieve neighboring phrases that are useful for translation even when overall sentence similarity is low.
Outcome:	The proposed approach performs well on a heterogeneous dataset with WMT, IWSLT, JRC-Acquis and OpenSubtitles.

Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction (2021.emnlp-main)

Copied to clipboard

Xuming Hu, Chenwei Zhang, Yawen Yang, Xiaohe Li, Li Lin, Lijie Wen, Philip S. Yu

Low-Resource NLP

Challenge:	Existing methods to extract relation facts from limited labeled corpora are laborintensive to obtain . Existing approaches use self-training to generate pseudo labels that will cause gradual drift problem or leverage meta-learning scheme which does not solicit feedback explicitly.
Approach:	They propose a Gradient Imitation Reinforcement Learning method to encourage pseudo label data to imitate gradient descent direction on labeled data and bootstrap its optimization capability through trial and error.
Outcome:	The proposed method handles two major scenarios in low-resource relation extraction when no unlabeled data is available.

Neural Compositional Denotational Semantics for Question Answering (D18-1)

Copied to clipboard

Nitish Gupta, Mike Lewis

Natural Language Interfaces

Challenge:	a new model for compositional questions is needed to answer multi-step reasoning . the model is inspired by formal approaches to compositional semantics .
Approach:	They propose an end-to-end differentiable model for interpreting compositional questions . they build a latent tree of interpretable expressions over a sentence .
Outcome:	The proposed model outperforms RNN encoders when test questions are longer than training questions.

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers (2023.findings-acl)

Copied to clipboard

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, Furu Wei

BBTv2: Towards a Gradient-Free Future with Large Language Models (2022.emnlp-main)

Challenge:	Large pretrained language models have shown surprising in-context learning ability . despite the great success in performance, its working mechanism remains unclear .
Approach:	They explain language models as meta-optimizers and understand in-context learning as implicit finetuning . they find that Transformer attention has a dual form of gradient descent .
Outcome:	The proposed model can predict labels for unseen inputs without parameter updates . the proposed model outperforms smaller models with a single parameter update .

Copied to clipboard

Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, Xipeng Qiu

Model Surgery: Modulating LLM’s Behavior Via Simple Parameter Editing (2025.naacl-long)

Challenge:	Recent work on parameter-efficient tuning (PET) only tunes a small portion of parameters while keeping most of the parameters of the LLM unchanged.
Approach:	They propose an improved version of Black-Box Tuning to tune PTMs through gradient descent . they prepend continuous prompts to every layer of the PTM and propose a divide-and-conquer gradient-free algorithm to optimize the prompts alternately.
Outcome:	The proposed method achieves comparable performance to full model tuning and state-of-the-art parameter-efficient methods under few-shot settings while maintaining much fewer tunable parameters.

Copied to clipboard

Huanqian Wang, Yang Yue, Rui Lu, Jingxin Shi, Andrew Zhao, Shenzhi Wang, Shiji Song, Gao Huang

Challenge:	Current approaches for detoxification or preventing jailbreaking involve fine-tuning billions of parameters through gradient descent with substantial computational cost.
Approach:	They propose to use supervised fine-tuning and Reinforcement Learning from human feedback to modify LLMs' behavior by directly editing a small subset of parameters.
Outcome:	Experiments show that editing a small subset of parameters can modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreak, with only inference-level computational resources.

Benchmarking Approximate Inference Methods for Neural Structured Prediction (N19-1)

Copied to clipboard

Lifu Tu, Kevin Gimpel

Learning Task Representations from In-Context Learning (2025.findings-acl)

Challenge:	Structured prediction models often involve complex inference problems for which finding exact solutions is intractable.
Approach:	They propose to perform gradient descent with respect to the output structure directly and train a neural network to perform inference.
Outcome:	The proposed methods achieve better speed/accuracy/search error trade-off than gradient descent while being faster than exact inference at similar accuracy levels.

Copied to clipboard

Baturay Saglam, Xinyang Hu, Zhuoran Yang, Dionysis Kalogerias, Amin Karbasi

Optimizing Test-Time Query Representations for Dense Retrieval (2023.findings-acl)

Challenge:	Existing methods for generalizing tasks to modalities beyond text fail to generalize effectively to linguistic tasks.
Approach:	They propose a method for encoding task information in ICL prompts as a function of attention heads within the transformer architecture.
Outcome:	The proposed method extracts task-specific information from in-context demonstrations and excels in both text and regression tasks.

Copied to clipboard

Mujeen Sung, Jungsoo Park, Jaewoo Kang, Danqi Chen, Jinhyuk Lee

Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (2021.naacl-main)

Challenge:	Recent developments of dense retrieval rely on quality representations of queries and contexts from pre-trained query and context encoders.
Approach:	They propose a test-time optimization of query representations that provides fine-grained pseudo labels over retrieval results.
Outcome:	The proposed algorithm improves open-domain question answering accuracy and direct re-ranking by up to 2.0% while running 1.3–2.4x faster with an efficient implementation.

Copied to clipboard

Guanghui Qin, Jason Eisner

Machine Unlearning of Pre-trained Large Language Models (2024.acl-long)

Challenge:	Pretrained language models retain factual knowledge that can be extracted with a sentential prompt.
Approach:	They propose to learn prompts by gradient descent, either fine-tuning prompts or starting from random initialization.
Outcome:	The proposed approach outperforms existing methods on English LMs and tasks.

Copied to clipboard

Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, Xiang Yue

Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight (2020.acl-main)

Challenge:	Using curated datasets, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining.
Approach:	They propose a framework for machine unlearning in pre-trained LLMs and integrate gradient ascent with gradient descent on in-distribution data to achieve robustness.
Outcome:	The proposed framework is over 105 times more efficient than retraining on in-distribution data and provides detailed guidelines for efficient hyperparameter tuning in the unlearning process.

Copied to clipboard

Hengyi Cai, Hongshen Chen, Yonghao Song, Cheng Zhang, Xiaofang Zhao, Dawei Yin

Dialogue Systems & Conversational Agents

Challenge:	Current state-of-the-art neural dialogue models learn from human conversations . however, due to the open-ended nature of human conversations, the quality of training data varies .
Approach:	They propose a data manipulation framework to augment and highlight effective training samples . they also propose to increase its manipulation skills through gradient descent with validation samples a reshaping framework to proactively restructure the data distribution towards reliable samples is also proposed .
Outcome:	The proposed framework improves the performance of open-domain neural dialogue models with respect to evaluation metrics and human judgments.

Efficient Adversarial Training with Robust Early-Bird Tickets (2022.emnlp-main)

Copied to clipboard

Zhiheng Xi, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang

Is In-Context Learning a Type of Error-Driven Learning? Evidence from the Inverse Frequency Effect in Structural Priming (2025.naacl-long)

Challenge:	Existing methods to improve the robustness of pre-trained language models are expensive because of the need to generate adversarial examples via gradient descent.
Approach:	They propose an adversarial optimization method that searches for robust tickets with structured sparsity in the early stage and fine-tunes tickets in the remaining time.
Outcome:	The proposed method achieves up to 7 13 training speedups while maintaining comparable or even better robustness compared to the most competitive state-of-the-art methods.

Copied to clipboard

Zhenghao Zhou, Robert Frank, R. Thomas McCoy

Psycholinguistics

Challenge:	Recent pre-trained large language models have shown the capacity to perform in-context learning (ICL) this capability could provide a way to bridge the divide between language models and humans.
Approach:	They propose a new way of diagnosing whether ICL is error-driven learning . they simulated structural priming with ICL and found the effect was stronger .
Outcome:	The proposed method is based on the inverse frequency effect (IFE) phenomenon is similar to error-driven learning in large language models .

A Symbolic Adversarial Learning Framework for Evolving Fake News Generation and Detection (2025.emnlp-main)

Copied to clipboard

Chong Tian, Qirong Ho, Xiuying Chen

Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts (2024.findings-acl)

Challenge:	Rapid LLM advancements heighten fake news risks by enabling the automatic generation of increasingly sophisticated misinformation.
Approach:	They propose a framework that implements an adversarial training paradigm by an agent symbolic learning optimization process rather than numerical updates.
Outcome:	The proposed framework generates sophisticated fake news that degrades state-of-the-art detection performance by 53.4% in Chinese and 34.2% in English on average.

Copied to clipboard

Ganesh Jawahar, Haichuan Yang, Yunyang Xiong, Zechun Liu, Dilin Wang, Fei Sun, Meng Li, Aasish Pappu, Barlas Oguz, Muhammad Abdul-Mageed, Laks Lakshmanan, Raghuraman Krishnamoorthi, Vikas Chandra

Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning (2023.acl-long)

Challenge:	Neural architecture search (NAS) uses weight-sharing supernets to generate diverse subnetworks without retraining.
Approach:	They propose a weight-sharing supernet that leverages mixture-of-experts to enhance supernet model expressiveness with minimal training overhead.
Outcome:	The proposed method achieves state-of-the-art (SoTA) performance in NAS for fast machine translation models, surpassing NAS-BERT and AutoDistil across various model sizes.

Copied to clipboard

Tianxiang Sun, Zhengfu He, Qin Zhu, Xipeng Qiu, Xuanjing Huang

Unraveling Feature Extraction Mechanisms in Neural Networks (2023.emnlp-main)

Challenge:	Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks.
Approach:	They propose to combine pre-trained modules with pre-trains to boost prompt tuning for few-shot learning.
Outcome:	The proposed model outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot learning settings.

Copied to clipboard

Xiaobing Sun, Jiaxi Li, Wei Lu

Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs (2024.findings-emnlp)

Challenge:	Neural networks have become indispensable across a variety of natural language processing tasks.
Approach:	They propose a theoretical approach based on Neural Tangent Kernels to investigate neural networks' internal mechanisms.
Outcome:	The proposed approach can be applied to analyze language modeling tasks . it shows that the choice of activation function can affect feature extraction .

Copied to clipboard

Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Lv Kaokao, Yi Liu

Differentiable Instruction Optimization for Cross-Task Generalization (2023.findings-acl)

Challenge:	Large Language Models (LLMs) have demonstrated exceptional proficiency in language-related tasks, but their deployment poses significant memory and storage requirements.
Approach:	They propose a method that optimizes rounding values and weight clipping within 200 steps.
Outcome:	The proposed method achieves exceptional results across 2 to 4 bits while maintaining low tuning costs and avoiding additional inference overhead.

Copied to clipboard

Masaru Isonuma, Junichiro Mori, Ichiro Sakata

Challenge:	Existing studies have shown that instruction tuning is effective for generalizing to arbitrary tasks unseen during training.
Approach:	They propose to introduce learnable instructions and optimize them with gradient descent to optimize instruction for generalization ability.
Outcome:	The proposed instruction extractor extracts appropriate instruction and improves generalization ability compared to manual instruction tuning.

Locally Differentially Private In-Context Learning (2024.lrec-main)

Copied to clipboard

Chunyan Zheng, Keke Sun, Wenhao Zhao, Haibo Zhou, Lixing Jiang, Shaoyang Song, Chunlai Zhou

Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer (2023.emnlp-main)

Challenge:	Large pretrained language models (LLMs) have shown surprising In-Context Learning ability.
Approach:	They propose a locally differentially private framework of in-context learning for LLMs that can be augmented with a private database for some specific task.
Outcome:	The proposed framework can predict labels without additional parameter modifications without input-label pairs .

Copied to clipboard

Ruize Gao, Zhirui Zhang, Yichao Du, Lemao Liu, Rui Wang